Image generation device and image generation method

ABSTRACT

Frame memories store captured images supplied from a plurality of imaging units, respectively. A motion detection portion detects the motion of a captured image of at least one of the imaging units. A mode determination portion sets, in accordance with at least either the amount of the motion detected by the motion detection portion or a power consumption limit of the image generation device, one of two modes, one for operating the plurality of imaging units simultaneously and the other for operating the plurality of imaging units intermittently. A control signal generation portion supplies, to the plurality of frame memories, a control signal adapted to control reading of the captured images from the plurality of frame memories in accordance with the mode set by the mode determination portion.

TECHNICAL FIELD

The present invention relates to an image generation device and imagegeneration method, and more particularly, to a device and method forcapturing a composite image.

BACKGROUND ART

With digital still cameras and digital video cameras widespread, thereare more occasions when captured still images and movies are stored incomputers for viewing and editing or displayed on screens of gameconsoles and television (TV) sets. Further, it is also popular to uploadcaptured movies to posting websites on the Internet for sharing withother users.

Some digital cameras are capable of capturing a panoramic image, makingit possible to capture a panoramic image with a wide view angle in ahandy manner. Software tools are in common use which generate apanoramic image by stitching together a plurality of images, shot indifferent shooting directions by digital cameras.

Further, a panoramic shooting mode or panoramic image synthesisapplication is available in some mobile devices such as camera-equippedmobile phones so that images, shot by tilting the mobile device atdifferent angles, are automatically synthesized into a panoramic image.

CITATION LIST Patent Literature [PTL 1] JP 2011-76249 A SUMMARYTechnical Problem

Panoramic shooting mode available with cameras and camera-equippedmobile devices develops a large error caused by positional deviationbecause images are shot while at the same time changing the shootingdirection of the camera or mobile device. Further, still images are shotone after another in different shooting directions for synthesis into apanoramic image, thus resulting in time lags between a plurality ofstill images shot in different shooting directions. Therefore, althoughit is possible to synthesize a panoramic still image, it istheoretically difficult to shoot a panoramic movie. For this reason, amethod is taken in which omnidirectional images are shot simultaneouslyusing a multiview camera incorporating a plurality of cameras, and thenthese images are synthesized into a panoramic photograph.

Such an omnidirectional multiview camera consumes a lot of power todrive a plurality of cameras. Further, large power is supplied tomemories to process as many pieces of image data as the number ofcameras. Still further, there is a problem that the memory capacity islarge to store image data for the number of cameras.

The present invention has been made in light of the foregoing, and it isan object of the present invention to provide a technology whichcontributes to dynamic reduction in power consumption withoutsignificantly deteriorating the movie image quality even when amultiview camera is used.

Solution to Problem

In order to solve the above problem, an image generation deviceaccording to a mode of the present invention includes a plurality offrame memories and a multiplexing section. The plurality of framememories store captured images from a plurality of imaging units. Themultiplexing section reads the captured images of the imaging units fromthe frame memories and multiplexes the captured images to generate animage stream. The multiplexing section includes a motion detectionportion, a mode determination portion, and a control signal generationportion. The motion detection portion detects the motion of a capturedimage of at least one of the imaging units. The mode determinationportion sets, in accordance with at least either the amount of themotion detected by the motion detection portion or a power consumptionlimit of the image generation device, one of two modes, one foroperating the plurality of imaging units simultaneously and the otherfor operating the plurality of imaging units intermittently. The controlsignal generation portion supplies, to the plurality of frame memories,a control signal adapted to control reading of the captured images fromthe plurality of frame memories in accordance with the mode set by themode determination portion.

Another mode of the present invention is an image generation method. Theimage generation method includes a step of storing captured images froma plurality of imaging units in a plurality of frame memories. The imagegeneration method further includes a multiplexing step of reading thecaptured images of the imaging units from the frame memories andmultiplexing the captured images to generate an image stream. Themultiplexing step includes a motion detection step, a mode determinationstep, and a control signal generation step. The motion detection stepdetects the motion of a captured image of at least one of the imagingunits. The mode determination step sets, in accordance with at leasteither the amount of the motion detected by the motion detection step ora power consumption limit, one of two modes, one for operating theplurality of imaging units simultaneously and the other for operatingthe plurality of imaging units intermittently. The control signalgeneration step supplies, to the plurality of frame memories, a controlsignal adapted to control reading of the captured images from theplurality of frame memories in accordance with the mode set by the modedetermination step.

It should be noted that any combinations of the above components and anyconversions of expressions of the present invention between “method,”“device,” “system,” “computer program,” “data structure,” “recordingmedium,” and so on are also effective as modes of the present invention.

Advantageous Effect of Invention

The present invention allows for easy generation of a synthesized imageusing a multiview camera.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a perspective view of a panoramic imaging device according toan embodiment.

FIG. 2 is a diagram describing the view angle of a side face imagingunit of a multiview imaging part and the installation position of anoperation/display unit of the panoramic imaging device shown in FIG. 1.

FIG. 3 is a diagram describing the internal structure of the multiviewimaging part shown in FIG. 1.

FIGS. 4( a) to 4(c) are diagrams describing the positional relationshipbetween six imaging units mounted to a fastening member shown in FIG. 3.

FIG. 5 is a diagram illustrating the central axes of the fasteningmember, to which the six imaging units are fastened, and a main bodypart.

FIGS. 6( a) and 6(b) are diagrams describing a configuration in whichthe multiview imaging part and the main body part are detachable fromeach other.

FIG. 7 is a diagram illustrating a configuration in which an imagingunit is provided also on the bottom face of the multiview imaging part.

FIGS. 8( a) to 8(d) are diagrams describing the relationship between theview angle of the imaging unit provided on one of the side faces of thefastening member and the view angle of the imaging unit provided on thezenith face of the fastening member.

FIG. 9 is a diagram illustrating, in a top view of the multiview imagingpart of the panoramic imaging device as seen from above, the overlapsbetween the fields of view of the five side face imaging units providedon the side faces of the fastening member.

FIGS. 10( a) and 10(b) are diagrams describing the operation screendisplayed on the operation/display unit.

FIG. 11 is a functional configuration diagram of a panoramic moviestream generator implemented in a main circuit of the main body part.

FIG. 12 illustrates a configuration relating to movie stream switchingand recording control handled by the panoramic movie stream generator.

FIG. 13 is a flowchart describing the steps for movie stream switchingand recording control handled by the panoramic movie stream generator.

FIGS. 14( a) and 14(b) are diagrams describing the data structure of apanoramic movie stream generated by movie stream switching and recordingcontrol.

FIG. 15 is a configuration diagram of a control system adapted toselectively control a multiview camera.

FIG. 16 is a diagram describing a functional configuration of themultiview camera control system shown in FIG. 15.

FIG. 17( a) is a flowchart describing a write process performed by themultiview camera control system shown in FIG. 16, and FIG. 17( b) is aflowchart describing a read process performed by the multiview cameracontrol system shown in FIG. 16.

FIG. 18 is a diagram describing an example of implementation of themultiview camera control system shown in FIG. 16.

FIG. 19 is a diagram describing another example of implementation of themultiview camera control system shown in FIG. 16.

FIGS. 20( a) and 20(b) are diagrams describing examples of the operationscreen displayed on the operation/display unit 40.

FIG. 21 is a diagram describing a case in which imaging units arecollectively set up by specifying a reference camera.

FIG. 22 is a diagram illustrating an example of the operation screenwhen the imaging units are collectively set up by specifying a referencecamera.

FIG. 23 is a diagram describing a case in which an outdoor image is shotfrom indoors through a window.

FIG. 24 is a diagram illustrating an example of the operation screenwhen an outdoor image is shot from indoors through a window.

FIG. 25 is a diagram illustrating an example of the operation screenwhen an outdoor image is shot from indoors through a window.

FIG. 26 is a diagram describing a case in which some of the cameras arefrozen.

FIG. 27 is a diagram illustrating an example of the operation screenwhen some of the cameras are frozen.

FIG. 28 is a functional configuration diagram of the panoramic moviestream generator adapted to perform multistep exposure shooting.

FIGS. 29( a) and 29(b) are diagrams describing captured images shot bysetting an identical reference exposure value in the plurality ofimaging units for comparison.

FIGS. 30( a) to 30(c) are diagrams describing the reference exposurevalue set in the imaging units by an exposure setting section shown inFIG. 28 and variable exposure values that are obtained by changing thereference exposure value in multiple steps in such a manner that thevariable exposure values are centered around the reference exposurevalue.

FIG. 31 is a diagram describing a panoramic image synthesized by an HDRsynthesis section shown in FIG. 28.

FIG. 32 is a flowchart describing the steps for conducting multistepexposure shooting.

FIGS. 33( a) to 33(c) are diagrams describing a method used by the HDRsynthesis section shown in FIG. 28 to proceed with pseudo HDR synthesis.

FIG. 34 is a functional configuration diagram of the panoramic moviestream generator capable of adjusting the azimuth.

FIGS. 35( a) and 35(b) are diagrams describing a user interface forsetting the shooting direction of the imaging unit in the frontdirection to magnetic north.

FIGS. 36( a) and 36(b) are diagrams describing a user interface forsetting the shooting direction of the imaging unit in the frontdirection to magnetic north.

FIGS. 37( a) and 37(b) are diagrams describing a user interface forsetting the shooting direction of the specific imaging unit to magneticnorth.

FIGS. 38( a) and 38(b) are diagrams describing a user interface forsetting the shooting direction of the specific imaging unit to magneticnorth.

DESCRIPTION OF EMBODIMENTS

[Configuration of Panoramic Imaging Device]

FIG. 1 is a perspective view of a panoramic imaging device 100 accordingto an embodiment. The panoramic imaging device 100 includes a multiviewimaging part 10 and a main body part 30. In the present embodiment, themultiview imaging part 10 and the main body part 30 are both cylindricalin shape and connected together with their central axes aligned witheach other.

The multiview imaging part 10 is covered with a cylindrical cameraenclosure 12. A plurality of imaging units 20A to 20F (hereinafterreferred to as the imaging units 20 when collectively referred to) areincorporated in the camera enclosure 12. Each of the imaging units 20Ato 20F includes components required for shooting such as lens andimaging element. In the present embodiment, the five imaging units 20Ato 20E (hereinafter also referred to as the “side face imaging units”)are arranged equiangularly so that their lenses are located on thecircumference of the cylindrical camera enclosure 12. The single imagingunit 20F (hereinafter also referred to as the “zenith imaging unit”) isarranged so that its lens is located on the top face of the cameraenclosure 12.

The main body part 30 includes an operation/display unit 40, a battery,a main circuit, and an external interface. The operation/display unit 40is provided at a position recessed inward from the outer circumferenceof the cylindrical main body part 30. Movie data captured by the imagingunits 20A to 20F of the multiview imaging part 10 is transmitted to themain circuit of the main body part 30 via a serial interface or USB(Universal Serial Bus) interface. The main circuit implements a functionof generating a panoramic movie stream by multiplexing movie framescaptured by the imaging units 20A to 20F. The main circuit includes anexternal interface used to connect an external recording device such asUSB memory. A generated panoramic movie stream is recorded to theexternal recording device via the external interface.

The operation/display unit 40 is a display with a touch panel. Theoperation/display unit 40 displays, on its display, each of the imagescaptured by one of the imaging units 20A to 20F of the multiview imagingpart 10. Further, the operation/display unit 40 can display, on itsdisplay, a panoramic image obtained by synthesizing a plurality ofimages captured by the plurality of imaging units 20A to 20F. Stillfurther, the operation/display unit 40 can display, on its display, anoperation menu so that a user can make an operation input to the displayscreen by directly touching the touch panel while watching the displayscreen.

FIG. 2 is a diagram describing the view angle of the side face imagingunit 20A of the multiview imaging part 10 and the installation positionof the operation/display unit 40 of the panoramic imaging device 100.The main body part 30 of the panoramic imaging device 100 is attached toa tripod 80.

The spread of the field of view shot by the side face imaging unit 20Ais indicated by a view angle θ of the side face imaging unit 20A asillustrated in FIG. 2. A photographer sits in front of the tripod 80 andextends his or her hand from under the side face imaging unit 20A, thusoperating the operation menu that appears on the operation/display unit40. The operation/display unit 40 is provided in the recessed area ofthe cylindrical main body part 30. Therefore, even when operating theon-screen operation menu by touching the touch panel, the photographercan touch the touch panel outside the field of view of the side faceimaging unit 20A. This prevents the photographer's fingers fromunintentionally appearing in an image.

Further, because the operation/display unit 40 is provided in therecessed area of the cylindrical main body part 30, it is possible toprevent illumination light emitted from the operation/display unit 40from appearing in any of the imaging units 20A to 20F. To reduce, to agiven level or less, the impact of illumination light from theoperation/display unit 40 appearing in the images, the depth of therecess in which the operation/display unit 40 is provided and theinclination of the operation/display unit 40 in the recessed area aredesigned.

FIG. 3 is a diagram describing the internal structure of the multiviewimaging part 10. The plurality of imaging units 20A to 20F are fastenedto a fastening member 14 inside the camera enclosure 12 of the multiviewimaging part 10. If the plurality of imaging units 20A to 20F werefastened to different fastening members, a positional displacement wouldvary from one imaging unit to another of the plurality of imaging units20A to 20F due to deformation of the fastening members caused, forexample, by thermal expansion. If the plurality of imaging units 20A to20F are fastened to the single fastening member 14 as in the presentembodiment, the impact exerted on the imaging units 20A to 20F by thecharacteristic change of the fastening member 14 due, for example, tothermal expansion is constant, thus contributing to improved positionalaccuracy between the plurality of imaging units 20A to 20F.

In the present embodiment, the fastening member 14 is in the shape of aregular pentagonal prism, with each of the five side face imaging units20A, 20B, 20C, 20D, and 20E provided on one of the side faces of thepentagonal prism, and the zenith imaging unit 20F provided on the zenithface of the pentagonal prism. Although in the shape of a pentagonalprism in the present embodiment, the fastening member 14 may be in theshape of other polygonal prism, with one imaging unit provided on eachof the side faces of the polygonal prism.

In general, subjects are located in 360 horizontal degrees in apanoramic image, which are often the sky above the head and the groundunder the feet. For this reason, if the imaging units are provided onthe outer circumferential faces of a polygonal prism as in the presentembodiment, and if one imaging unit is provided on the top face of thepolygonal prism, it is possible to shoot a plurality of images bychanging the direction of line of sight equiangularly in the horizontaldirection of the photographer. Further, the plurality of imaging unitsare arranged equidistantly. This makes it easy to conduct imagesynthesis which is required to generate a panoramic image by stitchingtogether image data obtained from the imaging units. Still further,unlike a conventional multiview camera which has an imaging unit on eachof the faces of a regular dodecahedron (12 imaging units in total), thepresent embodiment is advantageous over the conventional multiviewcamera in that as many images as necessary for synthesizing a panoramicimage can be captured using fewer imaging units.

FIGS. 4( a) to 4(c) are diagrams describing the positional relationshipbetween the six imaging units 20A to 20F mounted to the fastening member14 shown in FIG. 3. FIG. 4( a) is a top view as seen from the directionof the central axis of the fastening member 14 (from the z axisdirection shown in FIG. 3). FIG. 4( b) is a side view as seen from thedirection vertical to one of the side faces of the prism of thefastening member 14 to which the imaging unit 20A is attached (from thex axis direction shown in FIG. 3). FIG. 4( c) is a side view as seenfrom the y axis direction shown in FIG. 3 which is vertical to the x andz axis directions shown in FIG. 3.

As illustrated in the top view of FIG. 4( a), the five side face imagingunits 20A to 20E are arranged around the fastening member 14 in theshape of a pentagonal prism in such a manner as to form a circle whoseradius is L with an equal circumferential angle (72 degrees) on thehorizontal plane. The shooting direction of each of the side faceimaging units 20A to 20E is the radial direction of the circle. Asillustrated in the side views of FIGS. 4( b) and 4(c), the zenithimaging unit 20F provided on the zenith face of the fastening member 14and the side face imaging unit 20B provided on the side face of thefastening member 14 are arranged in such a manner as to form a circlewhose radius is L with a circumferential angle of 90 degrees on thevertical plane. The zenith imaging unit 20F and each of the side faceimaging units 20A, 20C, 20D, and 20E provided on one of the other sidefaces of the fastening member 14 are similarly arranged in such a manneras to form a circle whose radius is L with a circumferential angle of 90degrees. In other words, the six imaging units 20A to 20F are arrangedin such a manner as to form a sphere whose radius is L.

FIG. 5 is a diagram illustrating the central axes of the fasteningmember 14, to which the six imaging units 20A to 20F are fastened, andthe main body part 30. As illustrated in FIG. 5, the multiview imagingpart 10 is connected to the main body part 30 in such a manner that thecentral axes of the fastening member 14 and the main body part 30 arealigned with each other. Further, a tripod seat fastening hole 50 of themain body part 30 is provided in the central axis of the main body part30. As a result, the position of the tripod connected to the tripod seatfastening hole 50 is aligned with the central axis of the fasteningmember 14. This makes it easy to adjust the shooting position and thedirection of line of sight even when the main body part 30 installed tothe tripod is tilted or rotated.

FIGS. 6( a) and 6(b) are diagrams describing a configuration in whichthe multiview imaging part 10 and the main body part 30 are detachablefrom each other. As illustrated in FIG. 6( a), the multiview imagingpart 10 and the main body part 30 may be detached from each other sothat image data is converted in accordance with the interface standardsuch as USB (Universal Serial Bus) used for connection with a peripheraldevice, and so that the multiview imaging part 10 is connected to themain body part 30 that is located physically remotely using, forexample, an extendable cable. Alternatively, the multiview imaging part10 and the main body part 30 may be connected together via a detachablerelay cable adapted to extend the camera's signal line. Stillalternatively, the multiview imaging part 10 and the main body part 30may each have a wireless communication function and be wirelesslyconnected together. This allows for panoramic shooting even in aconfined area into which the main body part 30 cannot be carried bycarrying only the multiview imaging part 10 into that area. For example,the inside of a hole can be panoramically shot by inserting themultiview imaging part 10 into that hole.

On the other hand, the main body part 30 may be a mobile terminal suchas tablet PC or mobile phone so that the multiview imaging part 10 andthe main body part 30 are detached from each other for exchange of datathrough wireless communication. This allows the photographer to carrythe main body part 30 and operate the multiview imaging part 10 from aremote location, making it even easier to prevent the photographer fromaccidentally appearing in an image.

FIG. 6( b) illustrates a configuration in which a tripod seat fasteninghole 52 is provided on the bottom face of the multiview imaging part 10.This configuration allows a tripod to be directly connected to themultiview imaging part 10. If the multiview imaging part 10 has awireless communication function, data captured by the multiview imagingpart 10 can be transmitted to the main body part 30 or other mobileterminal.

FIG. 7 is a diagram illustrating a configuration in which an imagingunit 20G is provided also on the bottom face of the multiview imagingpart 10. The provision of the imaging unit 20G on the bottom face inaddition to the imaging unit 20F provided on the top face as describedabove allows for omnidirectional shooting. In this case, a tripod cannotbe connected to the bottom face. Therefore, shooting is conducted, forexample, by suspending the multiview imaging part 10 from the ceilingwith a wire. Alternatively, omnidirectional shooting may be conducted bythrowing the multiview imaging part 10 in the air. If the multiviewimaging part 10 has a wireless communication function, data captured bythe multiview imaging part 10 can be transmitted to the main body part30 or other mobile terminal.

FIGS. 8( a) to 8(d) are diagrams describing the relationship between theview angle of the imaging unit 20A provided on one of the side faces ofthe fastening member 14 and the view angle of the imaging unit 20Fprovided on the zenith face of the fastening member 14.

FIG. 8( a) illustrates the relationship between a field of view 60A ofthe side face imaging unit 20A provided on one of the side faces of thefastening member 14 and a field of view 60B of the zenith imaging unit20F provided on the zenith face of the fastening member 14. FIG. 8( b)is a top view of the multiview imaging part 10 of the panoramic imagingdevice 100 as seen from above (view as seen from the z axis direction inFIG. 3), illustrating a horizontal view angle θ_(AH) of the field ofview 60A of the side face imaging unit 20A provided on one of the sidefaces of the fastening member 14. FIGS. 8( c) and 8(d) are side views ofthe panoramic imaging device 100 as seen from side (views as seen fromthe x and y axis directions in FIG. 3). FIG. 8( c) illustrates ahorizontal view angle θ_(FH) of the zenith imaging unit 20F provided onthe zenith face of the fastening member 14. FIG. 8( d) illustrates avertical view angle θ_(AV) of the side face imaging unit 20A provided onone of the side faces of the fastening member 14 and a vertical viewangle θ_(FV) of the zenith imaging unit 20F provided on the zenith faceof the fastening member 14.

As an example, the horizontal view angle θ_(AH) of the side face imagingunit 20A is θ_(AH)=121 degrees. This permits panoramic shooting in 360horizontal degrees thanks to the five side face imaging units 20A to 20Eprovided on the side faces of the fastening member 14. On the otherhand, the vertical view angle θ_(AV) of the side face imaging unit 20Ais θ_(AV)=134 degrees. With this and the horizontal view angle θ_(FH)(=134 degrees) or vertical view angle θ_(FV) (=121 degrees) of thezenith imaging unit 20F combined, panoramic shooting is possible in 180vertical degrees.

It should be noted that the relationship between the vertical view angleθ_(AV) and the horizontal view angle θ_(AH) of the side face imagingunit 20A is θ_(AV)>θ_(AH) in this example. The horizontal view angle ofan ordinary lens is larger than the vertical view angle thereof. Itshould be kept in mind here that the imaging units 20A to 20E areprovided on the side faces of the fastening member 14 in a mannerrotated 90 degrees relative to the normal orientation in order toincrease the view angles of the side face imaging units 20A to 20E inthe vertical direction.

FIG. 9 is a diagram illustrating, in a top view of the multiview imagingpart 10 of the panoramic imaging device 100 as seen from above, theoverlaps between the fields of view of the five side face imaging units20A to 20E provided on the side faces of the fastening member 14. Thehorizontal view angle of each of the side face imaging units 20A to 20Eis, for example, 121 degrees. Combining the images shot by the five sideface imaging units 20A to 20E permits panoramic shooting in 360horizontal degrees.

FIGS. 10( a) and 10(b) are diagrams describing the operation screendisplayed on the operation/display unit 40. Images shot by the siximaging units 20A to 20F of the panoramic imaging device 100 shown inFIG. 10( a) are displayed respectively in areas A to F of the operationscreen shown in FIG. 10( b).

Selecting any of the areas A to F shown in FIG. 10( b) on the touchpanel allows one of the imaging units 20A to 20F to be selected as areference camera. Here, the area B is selected. As a result, the area Bis highlighted, for example, by displaying the frame of the area B in adifferent color. The area B is associated with the imaging unit 20B. Theimaging unit 20B is used as a reference for setting shooting conditionssuch as exposure and white balance.

When the photographer presses an auto exposure (AE) lock button 74, theimaging unit 20B, the reference camera, is set to an optimal exposurevalue by auto exposure. This optimal exposure value is reflected in theother imaging units 20A, 20C, 20D, 20E, and 20F. The AE lock button 74is a toggle button. When the photographer presses the AE lock button 74again, AE is unlocked, thus allowing automatic adjustment of theexposure of each of the imaging units 20A to 20F individually.

When the photographer presses an auto white balance (AWB) lock button 76next, the white balance of the imaging unit 20B, the reference camera,is adjusted by auto white balance. The correction value thereof isreflected in the other imaging units 20A, 20C, 20D, 20E, and 20F. TheAWB lock button 76 is also a toggle button. When the photographerpresses the AWB lock button 76 again, AWB is unlocked, thus allowingautomatic adjustment of white balance of each of the imaging units 20Ato 20F individually.

When an extension control button 78 is pressed, various setup menusappear, thus allowing further setting of shooting parameters.

When the photographer presses a movie recording start/stop button 70,movies are shot by the imaging units 20A to 20F. When the photographerpresses the movie recording start/stop button 70 again, the shooting ofmovies stops. When the photographer presses a still image shootingbutton 72, still images are shot by the imaging units 20A to 20F. Therecorded movies or still images are stored, for example, in a USB memoryconnected to the external interface of the main circuit of the main bodypart 30.

FIG. 11 is a functional configuration diagram of a panoramic moviestream generator 200 implemented in the main circuit of the main bodypart 30.

A shooting control section 210 sets shooting parameters such as exposurevalue, white balance value, and frame rate collectively in all theimaging units 20A to 20F of the multiview imaging part 10 orindividually in each of these units. Further, the shooting controlsection 210 controls the zooming of the imaging units 20A to 20F and thestart and stop of shooting.

Movie frames shot by the imaging units 20A to 20F are storedrespectively in frame memories 220A to 220F.

A movie stream multiplexing section 230 multiplexes the movie framesstored in the frame memories 220A to 220F and shot by the imaging units20A to 20F, generating a multiview movie stream and supplying the moviestream to a panoramic stitching section 242. The panoramic stitchingsection 242 stitches together the movie frames shot by the imaging units20A to 20F through a stitching process, synthesizing a panoramic movieand generating a panoramic movie stream. If power consumption islimited, the panoramic stitching section 242 can output a multiviewmovie stream as-is without performing a stitching process.Alternatively, the panoramic stitching section 242 can output both amultiview movie stream and a panoramic movie stream as necessary. Thepanoramic stitching section 242 records at least either a multiviewmovie stream or a panoramic movie stream to a panoramic movie storagesection 240.

A display control section 260 reads the movie frames shot by the imagingunits 20A to 20F from the frame memories 220A to 220F, displaying themovie frames on the screen of the operation/display unit 40.

A user interface section 250 supplies operation menu information of theimaging units 20A to 20F to the display control section 260, allowingthe display control section 260 to display the operation menu on thescreen of the operation/display unit 40.

A touch panel control section 270 detects a touch operation on the touchpanel by a user's finger, supplying the touched position and otherinformation to the user interface section 250. The user interfacesection 250 identifies the nature of operation selected by the user onthe operation menu from the touched position information supplied fromthe touch panel control section 270, transmitting an operation commandto the shooting control section 210. The shooting control section 210controls the imaging units 20A to 20F based on the operation commandsupplied from the user interface section 250.

[Movie Stream Switching and Recording Control]

A description will be given next of movie stream switching and recordingcontrol handled by the panoramic movie stream generator 200 of thepanoramic imaging device 100.

FIG. 12 illustrates a configuration relating to movie stream switchingand recording control handled by the panoramic movie stream generator200. A description will be given here of different configurations andoperations of the shooting control section 210 and the movie streammultiplexing section 230 shown in FIG. 11. The panoramic movie storagesection 240, the display control section 260, the user interface section250, and the touch panel control section 270, i.e., the components shownin FIG. 11, are omitted in FIG. 12.

In FIG. 12, dashed lines represent control signal lines, and bold linesrepresent image data transfer channels. As already described in FIG. 11,movie frames shot by the imaging units 20A to 20F are storedrespectively in the frame memories 220A to 220F. The movie streammultiplexing section 230 reads the movie frames stored in the framememories 220A to 220F and shot by the imaging units 20A to 20F,multiplexing the movie frames and generating a multiview movie stream.

The movie stream multiplexing section 230 has functions of a motiondetection portion 232, a mode determination portion 234, and a controlsignal generation portion 236 in addition to the function ofmultiplexing movie frames shot by the imaging units 20A to 20F.

The motion detection portion 232 detects motion vectors of movie framesA to F shot by the imaging units 20A to 20F, finding the sum of themagnitudes of the motion vectors of the movie frames A to F.Alternatively, the motion detection portion 232 may detect motion vectoror vectors of the movie frames shot by the specific one or more imagingunits of interest so as to find the sum of the magnitudes of the motionvectors of the one or more movie frames of interest.

If the sum of the magnitudes of the motion vectors calculated by themotion detection portion 232 is smaller than a given threshold, the modedetermination portion 234 sets the imaging units 20A to 20F to “imagingunit intermittent operation mode.” In this mode, the imaging units 20Ato 20F are operated one after another in an intermittent manner atintervals of a frame, thus switching to one of the imaging units 20A to20F every frame. Then, the captured image, output from the switchedimaging unit, is read from the associated frame memory, after which thecaptured images are multiplexed into a movie stream.

If the sum of the magnitudes of the motion vectors calculated by themotion detection portion 232 is equal to or greater than the giventhreshold, the mode determination portion 234 sets the imaging units 20Ato 20F to “imaging unit simultaneous operation mode.” In this mode, allthe imaging units 20A to 20F are operated simultaneously. Then, thecaptured images, output from the imaging units 20A to 20F, are read fromthe frame memories 220A to 220F, after which the captured images aremultiplexed into a movie stream.

The mode determination portion 234 supplies, to the shooting controlsection 210 and the control signal generation portion 236, informationabout the operation mode of the imaging units that has been set.

The shooting control section 210 supplies, to the imaging units 20A to20F, a control signal appropriate to the operation mode set by the modedetermination portion 234.

In “imaging unit intermittent operation mode,” the shooting controlsection 210 supplies, to the imaging units 20A to 20F, a control signaladapted to operate the imaging units 20A to 20F one after another in anintermittent manner at intervals of a frame. This eliminates the need tosupply power to the inactive imaging units, thus contributing to reducedpower consumption.

In “imaging unit simultaneous operation mode,” the shooting controlsection 210 supplies, to the imaging units 20A to 20F, a control signaladapted to operate all the imaging units 20A to 20F simultaneously. Inthis case, all the imaging units 20A to 20F are active, thus resultingin more power consumption. However, the synthesized panoramic image doesnot suffer degradation even when the subject moves vigorously or duringa scene change.

The control signal generation portion 236 supplies, to the framememories 220A to 220F, a control signal appropriate to the operationmode set by the mode determination portion 234. In “imaging unitintermittent operation mode,” the control signal generation portion 236supplies, to the frame memories 220A to 220F, a control signal adaptedto switch to one of the frame memories 220A to 220F every frame, enablereading from the switched frame memory and disable reading from allother frame memories. In “imaging unit simultaneous operation mode,” thecontrol signal generation portion 236 supplies, to the frame memories220A to 220F, a control signal adapted to enable reading from all theframe memories 220A to 220F.

Here, even in “imaging unit simultaneous operation mode,” if thebandwidth of the interface adapted to transfer image data is greaterthan (number of imaging units)×(data amount per frame)×(frame rate), noproblem will arise when the reading from all the frame memories 220A to220F is enabled and image data is read simultaneously from the framememories 220A to 220F. However, if the data amount of only one of theimaging units is permitted, image data may be temporarily recorded tothe frame memories 220A to 220F, after which the frame memories 220A to220F are switched from one to another to sequentially transmit imagedata.

Further, if intermittently operating the imaging units 20A to 20F posesa problem because the imaging units 20A to 20F are slow to start up, theimaging units 20A to 20F may be operated continuously while activatingthe frame memories 220A to 220F one after another in an intermittentmanner at intervals of a frame, thus switching to one of the framememories 220A to 220F every frame, reading the captured image from theswitched frame memory, and multiplexing the images into a movie stream.This also contributes to reduced power supplied to the frame memories.

Further, the mode determination portion 234 may switch between “imagingunit intermittent operation mode” and “imaging unit simultaneousoperation mode” in accordance with not only the amount of motion of thecaptured image but also the remaining battery charge or permissiblepower consumption of the panoramic imaging device 100. For example, inorder to operate the panoramic imaging device 100 for extended hours, itis necessary to keep the power consumption per unit time to a minimum.Therefore, the operation mode is switched to “imaging unit intermittentoperation mode.” On the other hand, if the remaining battery charge issufficient, the panoramic imaging device 100 may be operated in “imagingunit simultaneous operation mode” which is switched to “imaging unitintermittent operation mode” when the remaining battery charge goes low.Alternatively, if the panoramic imaging device 100 runs on a built-inchargeable battery, “imaging unit intermittent operation mode” may beselected. On the other hand, if the panoramic imaging device 100 runs onexternal power supplied from an AC adaptor, “imaging unit simultaneousoperation mode” may be selected.

The movie stream multiplexing section 230 multiplexes image frames,output from the frame memories 220A to 220F in either of the operationmodes, thus generating a multiview movie stream and supplying the moviestream to a main processor 280.

The main processor 280 performs a stitching process on the multiviewmovie stream, thus generating a panoramic movie stream and recording themovie stream to a secondary storage device 290 such as flash memory. Thefunctions of the panoramic stitching section 242 and a switching section246 are implemented by software in the main processor 280 to perform thepanoramic stitching process. These functions may be implemented byhardware.

The panoramic stitching section 242 stitches together image framesincluded in the multiview movie stream supplied from the movie streammultiplexing section 230 and shot by the imaging units 20A to 20F, thussynthesizing a panoramic image, generating a panoramic movie streamhaving chronological data of the panoramic image frame, and outputtingthe movie stream. If power consumption is limited in accordance with theremaining battery charge or the permissible power consumption, thepanoramic stitching section 242 outputs the multiview movie stream as-iswithout synthesizing any panoramic image.

The switching section 246 switches between the panoramic movie streamand multiview movie stream output from the panoramic stitching section242 in accordance with the power consumption limit, recording the moviestream to the secondary storage device 290. For example, the followingthree steps of control are possible in accordance with the magnitude ofavailable power.

(a) If the available power is small, the panoramic stitching section 242outputs a multiview movie stream without generating any panoramic moviestream. In this case, the switching section 246 turns OFF a switch 247of a data transfer channel 243 of the panoramic movie stream and turnsON a switch 248 of a data transfer channel 244 of the multiview moviestream, recording the multiview movie stream to the secondary storagedevice 290. This keeps power consumption to a minimum because nopanoramic stitching process is performed.

(b) If the available power is moderate, the panoramic stitching section242 generates and outputs a panoramic movie stream, but does not outputany multiview movie stream. In this case, the switching section 246turns ON the switch 247 of the data transfer channel 243 of thepanoramic movie stream and turns OFF the switch 248 of the data transferchannel 244 of the multiview movie stream, recording the panoramic moviestream to the secondary storage device 290. Because the panoramicstitching section 242 is active, more power is consumed than case (a).However, this option is advantageous in that a panoramic movie isgenerated in realtime.

(c) If the available power is large, the panoramic stitching section 242not only generates and outputs a panoramic movie stream but also outputsa multiview movie stream. In this case, the switching section 246 turnsON the switch 247 of the data transfer channel 243 of the panoramicmovie stream and turns ON the switch 248 of the data transfer channel244 of the multiview movie stream, recording the panoramic movie streamand the multiview movie stream to the secondary storage device 290. Botha multiview movie stream and a panoramic movie stream are generated andrecorded. This results in a high processing volume, more data to berecorded, and the highest power consumption. However, this option isadvantageous in that it can find more applications by recording not onlypanoramic movie streams but also multiview movie streams in advance.

It should be noted that although the frame rates of the imaging units20A to 20F are the same in the above description, the frame rate may bechanged between the imaging units 20A to 20F if the amount of motion isdifferent between the images captured by the imaging units 20A to 20F.In this case, the shooting control section 210 and the control signalgeneration portion 236 change, based on the ratio between the framerates, the ratio at which to switch between the imaging units 20A to 20Fand the frame memories 220A to 220F. That is, round robin control isperformed which assigns weights based on the ratio between the framerates.

FIG. 13 is a flowchart describing the steps for movie stream switchingand recording control handled by the panoramic movie stream generator200.

The motion detection portion 232 detects motion vectors from imagescaptured by the imaging units 20A to 20F (S10). The motion detectionportion 232 finds the sum of the magnitudes of the motion vectors of theimages captured by all the imaging units 20A to 20F or one or moreimaging units of interest (S12).

The mode determination portion 234 determines whether the sum of themagnitudes of the motion vectors of the captured images is equal to orgreater than a threshold (S14). If the sum of the magnitudes of themotion vectors of the captured images is smaller than the threshold (Nin S14), the mode determination portion 234 selects “imaging unitintermittent operation mode” (S20). In “imaging unit intermittentoperation mode,” “low frame rate recording mode” is selected in which amultiview movie stream or a panoramic movie stream is recorded at a lowframe rate.

If the sum of the magnitudes of the motion vectors of the capturedimages is equal to or greater than the threshold (Y in S14), the modedetermination portion 234 determines whether the panoramic imagingdevice 100 is operating within the power consumption limit (S16). Forexample, the mode determination portion 234 determines whether thepanoramic imaging device 100 is operating within the power consumptionlimit in consideration of permissible power consumption per unit timeand remaining battery charge in the case of shooting for extendedperiods. The mode determination portion 234 sets the panoramic imagingdevice 100 to “imaging unit intermittent operation mode” (S20) if thepanoramic imaging device 100 is not operating within the powerconsumption limit (N in S16). The mode determination portion 234 setsthe panoramic imaging device 100 to “imaging unit simultaneous operationmode” (S18) if the panoramic imaging device 100 is operating within thepower consumption limit (Y in S16). In “imaging unit simultaneousoperation mode,” “high frame rate recording mode” is selected in which amultiview movie stream or a panoramic movie stream is recorded at a highframe rate.

The shooting control section 210 switches the imaging units 20A to 20Fto simultaneous or intermittent operation in accordance with the modeset by the mode determination portion 234, and the control signalgeneration portion 236 switches the frame memories 220A to 220F tosimultaneous or intermittent output in accordance with the mode set bythe mode determination portion 234 (S22).

The panoramic movie stream generator 200 multiplexes frame data outputfrom the frame memories 220A to 220F in accordance with each mode,generating a multiview movie stream or a panoramic movie stream (S24).

If the shooting is stopped by the user (Y in S26), the movie streamswitching and recording control is terminated. If the shooting is notstopped by the user (N in S26), control returns to step S10 to repeatthe processes from step S10 onward.

FIGS. 14( a) and 14(b) are diagrams describing the data structure of amultiview movie stream generated by movie stream switching and recordingcontrol. Assume here that there are four imaging units, and that theframe rate of each of the imaging units is 30 fps (frames per second).

FIG. 14( a) illustrates a multiview movie stream generated in “imagingunit intermittent operation mode.” The imaging units 20A to 20D areoperated one after another in an intermittent manner at intervals of aframe, and frames A, B, C, and D captured respectively by the imagingunits 20A, 20B, 20C, and 20D are multiplexed in this order at intervalsof each frame. Each of the imaging units 20A to 20D operatesintermittently once every four times. As a result, the effective framerate of each of the imaging units 20A to 20D is 30 fps/4=7.5 fps. Thetransfer rate of a multiview movie stream after multiplexing isequivalent to 30 fps/4×4=30 fps. That is, a low frame rate multiviewmovie stream is generated. The four imaging units 20A to 20D operateintermittently. In the panoramic image obtained by synthesizing the fourimages captured by the four imaging units 20A to 20D, therefore, thecaptured images adjacent to each other are those captured at differenttimes. However, if there is not much motion, a panoramic image obtainedby stitching together the images that were captured at different timesdoes not suffer much degradation in quality.

FIG. 14( b) illustrates a multiview movie stream generated in “imagingunit simultaneous operation mode.” The imaging units 20A to 20D areoperated simultaneously, and the frames A, B, C, and D capturedrespectively by the imaging units 20A, 20B, 20C, and 20D are multiplexedat intervals of each frame. As a result, the effective frame rate ofeach of the imaging units 20A to 20D remains unchanged as 30 fps. Thetransfer rate of a multiview movie stream after multiplexing isequivalent to 30 fps×4=120 fps. That is, a high frame rate multiviewmovie stream is generated. The four imaging units 20A to 20D operatesimultaneously. In the panoramic image obtained by synthesizing the fourimages captured by the four imaging units 20A to 20D, therefore, thecaptured images adjacent to each other are those captured at the sametime. Therefore, a panoramic image obtained by stitching together theimages that were captured at the same time remains intact in qualityeven when the subject moves vigorously.

A description will be given below of the operational advantage of moviestream switching and recording control handled by the panoramic moviestream generator 200 described above.

In general, if the subject moves slowly or slightly, the image does notsuffer much degradation in quality even when the camera's frame rate isreduced. Conversely, if the subject moves vigorously, reducing thecamera's frame rate (equivalent to longer shutter speed in the case of aCMOS image sensor) leads to significant degradation in quality due tomotion blur. In the case of a multiview camera, simultaneously recordingall camera data at the highest frame rate provides the highest imagequality in any scene. However, this leads to more power consumption andclutters the storage capacity of the data storage device such as SSD(Solid State Drive) or HDD (Hard Disk Drive). For this reason, it ispreferable to be able to adaptively control the camera's operatingcondition and recorded data in accordance with the scene.

In movie stream switching and recording control according to the presentembodiment, it is possible to switch between the operation modes of theimaging units 20A to 20F and between the output modes of the framememories 220A to 220F in accordance with the motion of the scene. If thescene moves significantly, the image suffers degradation in qualityunless the effective frame rate of each imaging unit is secured.Therefore, “imaging unit simultaneous operation mode” is selected bygiving higher priority to image quality than power consumption. If thescene moves slightly, reducing the effective frame rate of each of theimaging units does not lead to significant degradation in image quality.Therefore, the operation mode is switched to “imaging unit intermittentoperation mode” to keep the power consumption to a minimum. As describedabove, switching between the operation modes in accordance with themotion of the scene keeps the power consumption to a minimum and reducesthe storage capacity without sustaining a significant decline in imagequality. Further, switching between the operation modes in accordancewith the remaining battery charge and the permissible power consumptionallows for efficient use of the panoramic imaging device 100. Stillfurther, a low frame rate image stream is generated in “imaging unitintermittent operation mode,” thus reducing the storage capacity.

[Multiview Camera Control System]

FIG. 15 is a configuration diagram of a control system adapted toselectively control a multiview camera. The shooting control section 210illustrated in FIG. 12 includes a bus interface/camera control interface300, a pass-through mask register 310, and a control data pass-throughlogic 320. The shooting control section 210 collectively and selectivelycontrols the arbitrary imaging units of the plurality of imaging units20A to 20F in response to an instruction from the main processor 280.For example, the shooting control section 210 can collectively orselectively control auto exposure (AE), auto white balance (AWB), andexposure value (EV).

The bus interface/camera control interface 300 is a bus interface of agiven standard adapted to connect the main processor 280 and the imagingunits 20 and a dedicated interface adapted to control the imaging units20A to 20F. The bus interface/camera control interface 300 supplies acontrol signal to the control data pass-through logic 320 in response toan instruction from the main processor 280.

The control data pass-through logic 320 is a circuit adapted to writecontrol data to the imaging units 20A to 20F or read data from theimaging units 20A to 20F. When writing data, the control datapass-through logic 320 can transmit write data to the imaging units 20Ato 20F by unicast, multicast, or broadcast. When reading data, thecontrol data pass-through logic 320 can receive read data from theimaging units 20A to 20F by unicast.

The pass-through mask register 310 is a register adapted to control theinput and output operations of the circuit of the control datapass-through logic 320. The pass-through mask register 310 is apass-through mask register that corresponds to the number of the imagingunits 20A to 20F.

FIG. 16 is a diagram describing a functional configuration of themultiview camera control system shown in FIG. 15. A camera controlinterface 302 is an example of the bus interface/camera controlinterface 300 shown in FIG. 15 and makes up an application processor 140together with the main processor 280. The application processor 140 isimplemented, for example, by system-on-chip (SoC) technology.

In the functional configuration diagram shown in FIG. 16, the controldata pass-through logic 320 shown in FIG. 15 is a circuit that connectsthe camera control interface 302 and the imaging units 20A to 20F usinga bus interface such as I2C or SPI. The control data pass-through logic320 includes switches 130A to 130F and a multiplexer 132.

A data transmitter Tx of the camera control interface 302 is connectedto the imaging units 20A to 20F via the switches 130A to 130F. Controldata specified by the main processor 280 is selectively written to theimaging units 20A to 20F. A control data write pass-through register 110(hereinafter simply referred to as the “write pass-through register”) isa pass-through mask register that corresponds to the number of theimaging units 20A to 20F. The write pass-through register 110 sets up amask to switch between ON and OFF states of the switches 130A to 130Fassociated respectively with the imaging units 20A to 20F.

For example, when the value of the write pass-through register 110 is‘110000,’ the switches 130A and 130B associated respectively with thetwo imaging units 20A and 20B are ON, and the switches 130C to 130Fassociated respectively with the remaining four imaging units 20C to 20Fare OFF. This allows control data to be selectively written to the twoimaging units 20A and 20B.

The imaging units 20A to 20F are connected to a data receiver Rx of thecamera control interface 302 via the multiplexer 132 having multipleinputs and a single output. Data is selectively read out from theimaging units 20A to 20F in accordance with a read instruction from themain processor 280. A camera setting value read pass-through register120 (hereinafter simply referred to as the “read pass-through register”)is a pass-through selection register that corresponds to the number ofthe imaging units 20A to 20F. The read pass-through register 120 sets upa mask to select one of a plurality of input streams of the multiplexer132.

For example, when the value of the read pass-through register 120 is‘010000,’ the input stream from the imaging unit 20B is selected fromamong the input streams from the six imaging units 20A to 20F that arefed to the multiplexer 132. As a result, the data from the imaging units20B is output from the multiplexer 132 as an output stream and suppliedto the receiver Rx of the camera control interface 302.

In the configuration shown in FIG. 16, only one interface is availablefor the main processor 280 to read or write data from or to the imagingunits 20A to 20F. It is possible to collectively or selectively read orwrite data from or to the imaging units 20A to 20F by simply rewritingthe pass-through masks of the write pass-through register 110 and theread pass-through register 120. Providing an independent interface foreach of the imaging units 20A to 20F leads to a larger circuit scale.However, the configuration shown in FIG. 16 has only one interface, thuscontributing to a reduced number of circuits. Further, it is possible toselect the desired imaging units to be written and simultaneously writedata to the selected imaging units by simply rewriting the writepass-through register 110, thus speeding up the write operation.

FIG. 17( a) is a flowchart describing a write process performed by themultiview camera control system shown in FIG. 16.

The main processor 280 prepares control data to be written to theimaging units 20A to 20F (S30). Control data is, for example, datarelating to shooting conditions such as exposure value and specified bythe user interface section 250 from the user.

The main processor 280 determines the target imaging units to be writtenand writes, to the write pass-through register 110, a value thatspecifies the imaging units to which control data is to be written(S32). If the write pass-through register 110 is a pass-through maskthat corresponds to the number of the imaging units, the writepass-through register 110 sets up a pass-through mask to select thetarget imaging units to be written. The ON/OFF states of the switches130A to 130F are set by the write pass-through register 110. The setupof a pass-through mask by the write pass-through register 110 isrequired to selectively transmit control data to the target imagingunits to be written.

If acknowledge (ACK) is required in response to write operation, themain processor 280 determines the target imaging units to be read andwrites, to the read pass-through register 120, a value that specifiesthe imaging units from which data is to be read (S34). If the readpass-through register 120 is a pass-through mask that corresponds to thenumber of the imaging units, the read pass-through register 120 sets upa pass-through mask to select the target imaging units to be read. Ofthe plurality of input streams of the multiplexer 132 having multipleinputs and a single output, the input streams specified by the readpass-through register 120 are selected and output as output streams. Thesetup of a pass-through mask by the read pass-through register 120 isrequired to selectively receive ACK from the target imaging units to bewritten.

Here, each of the target imaging units to be written returns ACK whenthe write operation is complete. Normally, ACK should be received fromall the target imaging units to be written, and the reception of ACKshould be verified individually. Such a process leads to a largercircuit scale and requires a procedure different from that of the busstandard. However, it is extremely rare that the write process failsunder normal circumstances. Therefore, ACK from only one of the targetimaging units is passed to verify the reception of ACK as an alternativeway to verify the reception of ACK from all the target imaging units.This also contributes to a simpler circuit. In the present embodiment,therefore, the read pass-through register 120 sets only one bit toverify the reception of ACK. In order to verify whether data has beenwritten to each of all the imaging units properly in a stricter manner,it is only necessary to individually read the register value for each ofthe imaging units to verify the reception of ACK.

The camera control interface 302 outputs, to the imaging units 20A to20F, a signal adapted to write control data from the transmissionterminal Tx via the switches 130A to 130F (S36). At this time, theON/OFF states of the switches 130A to 130F are changed in accordancewith the pass-through mask set up by the write pass-through register110, thus allowing control data to be written only to the imaging units20 selected as the target imaging units.

If ACK is required in response to write operation, the camera controlinterface 302 receives, from the imaging units 20A to 20F, an ACK signalwith the reception terminal Rx via the multiplexer 132 (S38). At thistime, ACK from the imaging units selected as target imaging units to beread is verified in accordance with the pass-through mask set up by theread pass-through register 120.

It should be noted that if ACK is not required in response to writeoperation, steps S34 and S38 can be omitted.

FIG. 17( b) is a flowchart describing a read process performed by themultiview camera control system shown in FIG. 16.

The main processor 280 determines the register of the imaging unit 20 tobe read (S40).

The main processor 280 determines the target imaging unit to be read andwrites, to the write pass-through register 110, a value that specifiesthe imaging units from which data is to be read (S42). If the writepass-through register 110 is a pass-through mask that corresponds to thenumber of the imaging units, the write pass-through register 110 sets upa pass-through mask to select the target imaging units to be read. TheON/OFF states of the switches 130A to 130F are set by the writepass-through register 110. The setup of a pass-through mask by the writepass-through register 110 is required to selectively transmit readaddresses to the target imaging units to be read.

Next, the main processor 280 determines the target imaging units to beread and writes, to the read pass-through register 120, a value thatspecifies the imaging units from which data is to be read (S44). If theread pass-through register 120 is a pass-through mask that correspondsto the number of the imaging units, the read pass-through register 120sets up a pass-through mask to select the target imaging units to beread. Of the plurality of input streams of the multiplexer 132 havingmultiple inputs and a single output, the input streams specified by theread pass-through register 120 are selected and output as outputstreams. The setup of a pass-through mask by the read pass-throughregister 120 is required to selectively receive data and ACK from thetarget imaging units to be read.

The camera control interface 302 outputs, to the imaging units 20A to20F, an imaging unit read address from the transmission terminal Tx viathe switches 130A to 130F (S46). At this time, the ON/OFF states of theswitches 130A to 130F are changed in accordance with the pass-throughmask set up by the write pass-through register 110, thus allowing theread address to be transmitted only to the imaging units 20 selected asthe target imaging units to be read.

The camera control interface 302 receives, from the imaging units 20A to20F, data for the specified address with the reception terminal Rx viathe multiplexer 132 (S48). At this time, data is received from theimaging units selected as the target imaging units to be read, and ACKfrom the imaging units selected as the target imaging units is verifiedin accordance with the pass-through mask set up by the read pass-throughregister 120.

FIG. 18 is a diagram describing an example of implementation of themultiview camera control system shown in FIG. 16. In the implementationexample shown in FIG. 18, a serial communication scheme forcommunication with peripheral devices called I2C (Inter-IntegratedCircuit) is used. A description will be given here of a case in whichthe four imaging units (imaging units 20A to 20D) are used forsimplicity.

An I2C interface 304 is provided on the side of the applicationprocessor 140 and serves as the I2C communication master. The imagingunits 20A to 20D are I2C communication slaves.

An I2C state monitoring circuit 150 is a circuit that enables a tristatebuffer 170 to receive in the receive state. As a result, data receptionfrom a multiplexer 176 is enabled. During write operation, the tristatebuffer 170 is blocked.

SDATA output from the I2C interface 304 is supplied to tristate buffers172A to 172D associated respectively with the imaging units 20A to 20D.The tristate buffer associated with the target imaging unit to bewritten is enabled in accordance with the pass-through mask of the writepass-through register 110, thus allowing the SDATA to be transmittedonly to the target imaging unit to be written. Pull-up resistors 174 areresistors that adjust the output signals of the tristate buffers 172A to172D to appropriate logic levels. SCLK output from the I2C interface 304is a clock signal adapted to synchronize the imaging units 20A to 20F.

Of the input terminals of the multiplexer 176 having multiple inputs anda single output, the input values from the input terminals specified inaccordance with the pass-through mask of the read pass-through register120 are multiplexed and supplied to the I2C interface 304 via thetristate buffer 170.

In FIG. 18, the pass-through mask of the write pass-through register 110is ‘1100.’ Therefore, control data is written to the imaging units 20Aand 20B during write operation. The pass-through mask of the readpass-through register 120 is ‘1000.’ Therefore, ACK is read from theimaging unit 20A during read operation.

FIG. 19 is a diagram describing another example of implementation of themultiview camera control system shown in FIG. 16. In the implementationexample shown in FIG. 19, a bus interface called SPI (Serial PeripheralInterface) is used which connects devices inside a computer. Here, adescription will be given also of a case in which the four imaging units(imaging units 20A to 20D) are used for simplicity.

An SPI interface 306 is provided on the side of the applicationprocessor 140 and serves as the SPI communication master. The imagingunits 20A to 20D are SPI communication slaves.

A CS# signal output from the SPI interface 306 is a chip select signaland supplied to NAND circuits 180A to 180D. The write pass-throughregister 110 is a pass-through mask adapted to select the target imagingunits 20A to 20D to be written. A register value is supplied to the NANDcircuits 180A to 180D that are respectively associated with the imagingunits 20A to 20D. In this example, the pass-through mask of the writepass-through register 110 is ‘1100.’ Therefore, ‘1’ is fed to the twoNAND circuits 180A and 180B, and ‘0’ is fed to the remaining two NANDcircuits 180C and 180D. Output values CAM1_CS# to CAM4_CS# of the NANDcircuits 180A to 180D are inverted and fed to the imaging units 20A to20D. As a result, the two imaging units 20A and 20B are selected,allowing control data TxData to be transmitted from MOSI (Master OutSlave In) of the SPI interface 306 to the imaging units 20A and 20B.

The SCLK signal from CLK of the SPI interface 306 is supplied to theimaging units 20A to 20F to synchronize these imaging units.

Read data RxData from a multiplexer 182 is received by MISO (Master InSlave Out) of the SPI interface 306. The read pass-through register 120is a pass-through mask adapted to select the target imaging units 20A to20D to be read. The read pass-through register 120 is used to select theinput signals of the multiplexer 182 having multiple inputs and a singleoutput. In this example, the pass-through mask of the read pass-throughregister 120 is ‘1000.’ Of the plurality of input streams of themultiplexer 182 having multiple inputs and a single output, the inputstream from the imaging unit 20A is selected and output as an outputstream of the multiplexer 182.

FIGS. 20( a) and 20(b) are diagrams describing examples of the operationscreen displayed on the operation/display unit 40. Images shot by thesix imaging units 20A to 20F of the panoramic imaging device 100 shownin FIG. 20( a) are displayed respectively in areas A to F of theoperation screen shown in FIG. 20( b).

The frame of each of the areas A to F of the operation screen appears,for example, in blue when selected as a reference camera. Further, whenthese areas are divided into groups, the frame of each group appears ina different color. The selection of a reference camera is toggled on oroff. Each click toggles between selection and deselection.

The movie recording start/stop button 70, the still image shootingbutton 72, the auto exposure lock button 74, the auto white balance lockbutton 76, and the extension control button 78 are the same as thosedescribed in FIG. 10( b). Therefore, the description thereof is omitted.

When a camera freeze button 75 is pressed, the selected imaging unitfreezes with the currently captured image. That is, the imaging unitremains operational and maintains, on screen, the same image as when thecamera freeze button 75 is pressed. The camera freeze button 75 is atoggle button. When the camera freeze button 75 is pressed again, thecamera unfreezes, thus resuming shooting by the selected imaging unit.

When a grouping button 77 is pressed, the areas arbitrarily selectedfrom among the plurality of areas A to F are grouped. In other words,the specific imaging units are selected and grouped from among theimaging units 20A to 20F. The desired areas to be grouped can beselected by clicking those areas or enclosing them with a freehand curveon the operation screen.

FIG. 21 is a diagram describing a case in which the imaging units arecollectively set up by specifying a reference camera. FIG. 22illustrates an example of the operation screen in that case.

As illustrated in FIG. 21, the imaging unit 20A is specified as areference camera so that the shooting parameters such as referencecamera's exposure value are set collectively in the other imaging units20B to 20F. In the case of shooting indoors, for example, the imagingunit 20A in the front direction is specified as a reference camera, andthe other imaging units 20B to 20F proceed with shooting with the sameexposure setting as that of the reference camera.

As illustrated in FIG. 22, when the auto exposure (AE) lock button 74 orthe auto white balance (AWB) lock button 76 is pressed with the area A,associated with the imaging unit 20A, selected as a reference camera,the same parameter is reflected in the other imaging units 20B to 20F.

The read pass-through register 120 is set to a value that specifies theimaging unit 20A selected as a reference camera as a target imaging unitto be read. As a result, the shooting parameters such as exposure valueand white balance value are read from the imaging unit 20A, thereference camera. Then, the write pass-through register 110 is set to avalue that specifies the remaining imaging units 20B to 20F other thanthe imaging unit 20A selected as a reference camera as target imagingunits to be written. As a result, the shooting parameters read from theimaging unit 20A, the reference camera, are collectively written to theremaining imaging units 20B to 20F.

A more specific operation procedure is as follows:

(1) Click the area A of the front camera on the operation screen. As aresult, the frame of the area A turns blue, and the imaging unit 20Aserves as a reference camera.

(2) Press the auto white balance lock button 76 to lock the whitebalance of all the imaging units 20A to 20F to the adjustment value ofthe imaging unit 20A, the reference camera.

(3) Press the auto exposure lock button 74 to lock the exposure value ofall the imaging units 20A to 20F to the adjustment value of the imagingunit 20A, the reference camera.

(4) Shoot with the movie recording start/stop button 70 or the stillimage shooting button 72.

FIG. 23 is a diagram describing a case in which an outdoor image is shotfrom indoors through a window. FIGS. 24 and 25 illustrate examples ofthe operation screen in that case.

As illustrated in FIG. 23, when an outdoor image is shot through awindow 190, the imaging units 20A, 20B, and 20E on the window sidenarrow down the exposure for outdoor shooting. However, it is necessaryfor the remaining imaging units 20C and 20D to shoot with an exposuresetting for indoor shooting.

As illustrated in FIG. 24, the areas A, B, and E associated with theimaging units 20A, 20B, and 20E on the window side are enclosed with afreehand curve 192 for selection. The grouping button 77 is pressed togroup the selected areas A, B, and E.

Next, of the areas A, B, and E that belong to the same group, the area Bis selected as illustrated in FIG. 25 so that the imaging unit 20B isselected as a reference camera. As a result, the same shooting parameteras for the imaging unit 20B, the reference camera, is reflected in theother imaging units 20A and 20E that belong to the same group.

The read pass-through register 120 is set to a value that specifies theimaging unit 20B selected as a reference camera in the group as a targetimaging unit to be read. As a result, the shooting parameters such asexposure value and white balance value are read from the imaging unit20B, the reference camera. Then, the write pass-through register 110 isset to a value that specifies the remaining imaging units 20A and 20Eother than the imaging unit 20B selected as a reference camera in thegroup as target imaging units to be written. As a result, the shootingparameters read from the imaging unit 20B, the reference camera, arecollectively written to the remaining imaging units 20A and 20E.

A more specific operation procedure is as follows:

(1) Select the areas A, B, and E associated with the imaging units 20A,20B, and 20E on the window side on the operation screen.

(2) Press the grouping button 77 to group the selected areas A, B, andE. As a result, the frames of the areas A, B, and E turn green, thusforming a first group. The remaining areas C, D and F form a secondgroup.

(3) Click the area B. As a result, the frame of the area B turns blue.The imaging unit 20B associated with the area B serves as a referencecamera of the green first group.

(4) Click the area D. As a result, the frame of the area D turns blue.The imaging unit 20D associated with the area D serves as a referencecamera of the second group.

(5) When the auto white balance lock button 76 is pressed, the whitebalance of the imaging units 20A, 20B, and 20E that belong to the firstgroup is locked to the adjustment value of the imaging unit 20B, thereference camera. Further, the white balance of the imaging units 20C,20D and 20F that belong to the second group is locked to the adjustmentvalue of the imaging unit 20D, the reference camera.

(6) When the auto exposure lock button 74 is pressed, the exposurevalues of the imaging units 20A, 20B, and 20E that belong to the firstgroup are locked to the adjustment value of the imaging unit 20B, thereference camera. Further, the exposure values of the imaging units 20C,20D and 20F that belong to the second group are locked to the adjustmentvalue of the imaging unit 20D, the reference camera.

(7) Shoot with the movie recording start/stop button 70 or the stillimage shooting button 72.

FIG. 26 is a diagram describing a case in which some of the cameras arefrozen. FIG. 27 illustrates an example of the operation screen in thatcase.

If a person 194 comes in during panoramic image shooting as illustratedin FIG. 26, the camera images are frozen with the person 194 notappearing in the images, and the camera shooting is temporarilysuspended. In this example, the shooting by the imaging units 20A and20B is temporarily suspended so that the person 194 does not appear inthe captured images, and the images not showing the person 194 arefrozen.

As illustrated in FIG. 27, select the desired areas A and B to freeze.Next, press the grouping button 77 to group the selected areas A and B.Then, select the area A or B as a reference camera and press the camerafreeze button 75. This makes it possible to temporarily stop the imagingunits 20A and 20B associated with the areas A and B that belong to thesame group and freeze the images with no person appearing therein.

A more specific operation procedure is as follows:

(1) Select the desired areas A and B to freeze on the operation screen.

(2) Press the grouping button 77 to group the selected areas A and B. Asa result, the frames of the areas A and B turn green.

(3) Click the area A or B. As a result, the imaging unit associated withthe selected area serves as a reference camera. Actually, however, thishas the same effect as specifying a group of cameras to be frozen.

(4) With no person appearing in the images, press the camera freezebutton 75. As a result, the group of cameras specified in step (3) isfrozen, thus causing the areas A and B to be frozen with images showingno person therein.

(5) From here onward, shoot with the remaining imaging units 20C to 20Fthat are not frozen.

As described above, in the multiview camera control system according tothe present embodiment, a mask register for writing data and a selectionregister for reading data are inserted between two interfaces, one usedby the main processor to access the cameras and another being a cameramodule interface. This provides collective and fast camera control, thuscontributing to easy and efficient camera control. A multiview cameracontrol system can be achieved simply by inserting registers between theexisting interfaces. This eliminates the need to change the cameramodules or processor interface, thus making it easy to design a controlsystem.

[Multistep Exposure Shooting]

FIG. 28 is a functional configuration diagram of the panoramic moviestream generator 200 adapted to perform multistep exposure shooting.Here, the components related to multistep exposure shooting are shown,and all other components are not shown for reasons of convenience. Anexposure setting section 400 shown in FIG. 28 can be implemented in theshooting control section 210 described with reference to FIG. 11. An HDRsynthesis section 420 shown in FIG. 28 can be implemented in the moviestream multiplexing section 230 described with reference to FIG. 11.

The exposure setting section 400 sets an exposure value in each of theplurality of imaging units 20A to 20F. More specifically, the exposuresetting section 400 sets an individual reference exposure value in eachof the imaging units 20A to 20F and further sets relative exposurevalues therein. The relative exposure values are centered around thereference exposure value and obtained by increasing or reducing thereference exposure value. The individual reference exposure value ofeach of the imaging units 20A to 20F may be set to an optimal value bythe auto exposure (AE) mechanism of each of the imaging units 20A to20F.

The exposure setting section 400 may set an exposure value in each ofthe imaging units 20A to 20F collectively or selectively using themultiview camera control system described with reference to FIG. 15.

The low dynamic range (LDR) captured images shot by the imaging units20A to 20F with varying exposure values are stored in frame buffers 430Ato 430F. Each of the frame buffers 430A to 430F is a frame memory thathas a storage area to store images shot with a plurality of exposurevalues set in the imaging units 20A to 20F.

The HDR synthesis section 420 reads the low dynamic range capturedimages shot by the imaging units 20A to 20F with varying exposure valuesfrom the frame buffers 430A to 430F, synthesizing the captured areaswith optimal exposure values through a stitching process and generatinga high dynamic range (HDR) panoramic image. It should be noted that ifthe ultimate output format is an LDR panoramic image, the HDR synthesissection 420 can generate an LDR panoramic image with a compresseddynamic range by subjecting the HDR synthesized image to a tone mappingprocess.

The HDR synthesis section 420 extracts feature points between adjacentcaptured images, synthesizing a panoramic image by stitching togetherthe adjacent captured images based on the feature points. A knowntechnology applied for image matching can be used for the feature pointextraction process. The panoramic image generated by the HDR synthesissection 420 is stored in a panoramic image storage section 410.

FIGS. 29( a) and 29(b) are diagrams describing captured images shot bysetting an identical reference exposure value in the plurality ofimaging units 20A to 20F for comparison.

FIG. 29( a) illustrates images 440A to 446A obtained by bracket shootingwith the imaging unit 20A in which a reference exposure value (0 in thiscase) is changed in multiple steps in such a manner that the varyingexposure values are centered around the reference exposure value. Theimage 442A is an example of an image shot with a reference exposurevalue EV=0. The images 443A, 444A, 445A, and 446A are examples of imagesshot respectively by adding 1, 4, 5, and 7 to the reference exposurevalue EV=0. On the other hand, the images 441A and 440A are examples ofimages shot respectively by subtracting 4 and 7 from the referenceexposure value EV=0. Adding 1 to the exposure value EV doubles theamount of light, whereas subtracting 1 from the exposure value EV halvesthe amount of light.

In FIG. 29( a), the image 440A with the exposure value EV of −7 has a“blocked up shadow,” thus resulting in a total lack of information. Theimage 446A with the exposure value EV of 7 partly has a “blown outhighlight.”

FIG. 29( b) illustrates images 440B to 446B obtained by bracket shootingwith the other imaging unit 20B in which the reference exposure value EVsame as that in the case of FIG. 29( a) is changed in multiple steps insuch a manner that the varying exposure values are centered around thereference exposure value. The image 442B is an example of an image shotwith the reference exposure value EV=0. The images 443B, 444B, 445B, and446B are examples of images shot respectively by adding 1, 4, 5, and 7to the reference exposure value EV=0. On the other hand, the images 441Band 440B are examples of images shot respectively by subtracting 4 and 7from the reference exposure value EV=0.

The images 440B and 441B respectively with the exposure values EV of −7and −4 have a “blocked up shadow,” thus resulting in a total lack ofinformation. The image 446B with the exposure value EV of 7 partly has a“blown out highlight.”

As described above, setting the same reference exposure value in all theimaging units 20A to 20F and increasing or reducing the referenceexposure value for bracket shooting leads to “blocked up shadows” or“blown out highlights” in the shot images, thus resulting in a waste ofimages.

With a multiview camera, the appropriate exposure value variessignificantly depending on the camera's field of view. In order toachieve an appropriate range of exposure in all directions, bracketshooting is effective which changes the exposure value in multiple stepsfor shooting. However, the dynamic range of brightness in the field ofview of a multiview camera varies significantly depending on the cameraangle. As a result, it is necessary to set the exposure values of allthe cameras in multiple steps in a wide range of exposure that spansfrom the darkest field of view to the brightest one for bracketshooting, thus resulting in more image data and more shooting time.Further, increasing or reducing the exposure value relative to the samereference exposure value for all the cameras leads to a large number ofimages having “blocked up shadows” or “blown out highlights” dependingon the camera angle, resulting in a waste of many images.

For this reason, in multistep exposure shooting according to the presentembodiment, an individual reference exposure value is set in each of theimaging units 20A to 20F, and the exposure value of each of the imagingunits 20A to 20F is varied in multiple steps by increasing or reducingthe individual reference exposure value for bracket shooting.

FIGS. 30( a) to 30(c) are diagrams describing the reference exposurevalue set in each of the imaging units 20A to 20F by the exposuresetting section 400 and relative exposure values that are obtained bychanging the reference exposure value in a predetermined number of stepsin such a manner that the relative exposure values are centered aroundthe reference exposure value. FIG. 30( a) illustrates a referenceexposure value EVa set in the imaging unit 20A and relative exposurevalues obtained by increasing the reference exposure value EVa in threesteps and reducing the reference exposure value EVa in three steps. FIG.30( b) illustrates a reference exposure value EVb set in the imagingunit 20B and relative exposure values obtained by increasing thereference exposure value EVb in three steps and reducing the referenceexposure value EVb in three steps. FIG. 30( c) illustrates a referenceexposure value EVc set in the imaging unit 20C and relative exposurevalues obtained by increasing the reference exposure value EVc in threesteps and reducing the reference exposure value EVc in three steps.

The reference exposure values EVa, EVb, and EVc set respectively in theimaging units 20A, 20B, and 20C are optimal exposure values in thefields of view of the imaging units 20A, 20B, and 20C which aregenerally different. Here, there is a difference of two steps betweenthe reference exposure values EVa and EVb set respectively in theimaging units 20A and 20B and between the reference exposure values EVband EVc set respectively in the imaging units 20B and 20C. Multistepexposure shooting is conducted by adding 3, 2, and 1 to and subtracting1, 2 and 3 from the reference exposure values EVa, EVb, and EVc, eachhaving a difference of two steps from its adjacent value. As a result,shooting is conducted using 11 different exposure values in total forthe three imaging units 20A, 20B, and 20C.

The number of steps for bracket shooting for each of the imaging units20 should preferably be determined so that the exposure ranges of theimaging units 20 with adjacent shooting directions partly overlap. Thisis designed to achieve a correlation in terms of exposure between theadjacent captured images during HDR synthesis.

Each of the imaging units 20A, 20B, and 20C performs multistep exposureshooting with an optimal reference exposure value and six more exposurevalues, three obtained by increasing the reference exposure value inthree steps and other three obtained by reducing the reference exposurevalue in three steps so that the six more exposure values are centeredaround the reference exposure value. This prevents “blocked up shadows”and “blown out highlights,” described with reference to FIGS. 29( a) and29(b), from appearing in images captured by the imaging units 20A, 20B,and 20C, thus eliminating a waste of images in multistep exposureshooting. Further, in contrast to 11-step multistep exposure shootingconducted by all the imaging units 20A, 20B, and 20C as hitherto beendone, bracket shooting is conducted by the imaging units 20A, 20B, and20C with only seven steps from +3 to −3 including a reference exposurevalue, thus contributing to significantly shorter time for bracketshooting. As described above, the present embodiment provides asignificantly larger dynamic range during HDR synthesis despite having asmall number of steps for bracket shooting conducted by the imagingunits 20A to 20F, thus allowing waste-free and efficient HDR panoramicimage synthesis.

FIG. 31 is a diagram describing a panoramic image synthesized by the HDRsynthesis section 420. The panoramic image shown in FIG. 31 is obtainedby stitching together the captured areas with appropriate exposurevalues of the images shot with a plurality of exposure values by theimaging units 20A to 20F for synthesis. For example, an area 450 is ascene outside the window which was extracted from a captured image shotby the imaging unit 20A with an appropriate exposure value for the fieldof view of the imaging unit 20A. An area 451 is an indoor scene whichwas extracted from a captured image shot by the imaging unit 20B with anappropriate exposure value for the field of view of the imaging unit20B.

FIG. 32 is a flowchart describing the steps for conducting multistepexposure shooting according to the present embodiment.

Select auto exposure (AE) and auto white balance (AWB) for the imagingunits 20A to 20F to proceed with shooting with an optimal referenceexposure value for each of the imaging units 20A to 20F (S50).

The exposure setting section 400 sets, in each of the imaging units 20Ato 20F, N positive and N negative relative exposure values which arecentered around the reference exposure value (S52).

Conduct bracket shooting with the N positive and N negative relativeexposure values using the imaging units 20A to 20F (S54).

The HDR synthesis section 420 stitches together the appropriate exposureareas of the captured images by bracket shooting using the imaging units20A to 20F, thus synthesizing a high dynamic range panoramic image(S56).

In the description given above, multistep exposure shooting is performedby increasing and reducing the reference exposure value in a givennumber of steps. Appropriate exposure areas of the captured images shotby the imaging units 20A to 20F through multistep exposure shooting arestitched together, thus synthesizing a high dynamic range panoramicimage. As a modification example, it is possible to synthesize a pseudohigh dynamic range panoramic image from images captured with anindividual reference exposure value set in each of the imaging units 20Ato 20F. A description will be given below of how to proceed with thispseudo HDR synthesis.

FIGS. 33( a) to 33(c) are diagrams describing a method used by the HDRsynthesis section 420 to perform pseudo HDR synthesis.

In pseudo HDR synthesis, each of the imaging units 20A to 20F isindividually controlled by auto exposure control and auto white balancecontrol to shoot a single image with the optimal exposure value. The HDRsynthesis section 420 synthesizes an HDR panoramic image by matchingcorresponding points between two captured images whose shootingdirections are adjacent to each other. At this time, we can assume thatthe luminance and color of the matching points are the same. Further,the optimal exposure values set in the imaging units 20A to 20F by autoexposure are known. Therefore, HDR synthesis is performed withoutrounding the luminance and color information of each of the pixels ofthe captured images to eight bits.

More specifically, assume that, because of the difference in optimalexposure value between the three imaging units 20A, 20B, and 20C, theeight-bit luminance information of the image shot by the imaging unit20A is in the luminance range shown in FIG. 33( a), the eight-bitluminance information of the image shot by the imaging unit 20B is inthe luminance range shown in FIG. 33( b), and the eight-bit luminanceinformation of the image shot by the imaging unit 20C is in theluminance range shown in FIG. 33( c).

That is, in a 32-bit high dynamic range, the luminance of the image shotby the imaging unit 20B is in a range that is three bits brighter thanthe luminance of the image shot by the imaging unit 20A. Further, theluminance of the image shot by the imaging unit 20C is in a range thatis six bits brighter than the luminance of the image shot by the imagingunit 20B. In this case, the pixels of the image shot by the imaging unit20B are shifted by three bits relative to the pixels of the image shotby the imaging unit 20A. The pixels of the image shot by the imagingunit 20C are shifted by nine bits (3+6 bits) relative to the pixels ofthe image shot by the imaging unit 20A. Then, the pixel values are addedup in the 32-bit range. As a result, it is possible to obtain asubstantially 17 (8+3+6)-bit high dynamic range pixel values.

It is also possible to obtain high dynamic range color differenceinformation by bit-shifting color difference information in accordancewith the optimal exposure value of each of the imaging units 20A to 20Fand then adding up the pixel values in the 32-bit space.

[Azimuth Setting]

In the description given above, the shooting directions of the imagingunits were not set to a specific direction. However, if the shootingdirection of the imaging unit 20A in the front direction is set tomagnetic north, for example, the center of the stitched panoramic imagefaces north. A description will be given below of the configurationadapted to set the azimuth of a panoramic image and the processperformed to do so.

FIG. 34 is a functional configuration diagram of the panoramic moviestream generator 200 capable of setting the azimuth. A description willbe given below of the configuration and operation different from thoseof the panoramic movie stream generator 200 shown in FIG. 11.

The multiview imaging part 10 or the main body part 30 of the panoramicimaging device 100 incorporates a triaxial magnetometer 252 and atriaxial acceleration sensor 254. Further, a triaxial gyro sensor mayalso be incorporated. The triaxial magnetometer 252 detects terrestrialmagnetism vectors in three axes. The triaxial acceleration sensor 254detects acceleration vectors of the panoramic imaging device 100 inthree axes.

Theoretically, so long as at least one of the triaxial gyro (angularspeed) sensor, the triaxial acceleration sensor, and the triaxialmagnetometer is incorporated in the panoramic imaging device 100,posture information can be acquired by detecting the inclination of thepanoramic imaging device 100 in three axes. Further, azimuth informationcan be acquired by detecting terrestrial magnetism vectors in three axesby the triaxial magnetometer.

Azimuth information acquired by the triaxial magnetometer 252 issupplied to the user interface section 250. Posture information acquiredby at least one of the triaxial magnetometer 252, the triaxialacceleration sensor 254, and the triaxial gyro sensor (if incorporated),is supplied to the user interface section 250.

The user interface section 250 supplies, to the display control section260, an instruction to display the azimuth in which the multiviewimaging part 10 of the panoramic imaging device 100 is shooting and theposture of the panoramic imaging device 100 on the screen. The displaycontrol section 260 displays information about the shooting azimuth andthe posture in graphics on the operation/display unit 40. Further, theuser interface section 250 causes, for example, an instruction to appearon the screen. The instruction prompts the user to adjust the directionof the multiview imaging part 10 so that the shooting direction of oneof the imaging units is magnetic north.

FIGS. 35( a) and 35(b) and FIGS. 36( a) and 36(b) are diagramsdescribing a user interface adapted to set the shooting direction of theimaging unit 20A in the front direction to magnetic north.

FIG. 35( a) is a schematic diagram of the multiview imaging part 10 ofthe panoramic imaging device 100 as seen from the zenith. The imagingunit 20A adapted to shoot in the front direction is located on theopposite side of the operation/display unit 40. Guidance is displayed onthe operation/display unit 40 to prompt the user to adjust the directionof the multiview imaging part 10 so that the shooting direction of theimaging unit 20A serving as a front camera is set to magnetic north.Magnetic north measured by the triaxial magnetometer 252 is indicated byan arrow in FIG. 35( a). There is a discrepancy between the shootingdirection of the imaging unit 20A and magnetic north.

FIG. 35( b) is a diagram describing the screen that appears on theoperation/display unit 40. Images shot by the six imaging units 20A to20F are normally displayed respectively in the areas A to F of theoperation screen of the operation/display unit 40 as shown in FIG. 10(b). However, when the area A associated with the imaging unit 20A, afront camera, is selected, only the image shot by the imaging unit 20Ais displayed in a large size on the operation/display unit 40 asillustrated in FIG. 35( b).

Under the area A (reference sign 42) in which the image shot by theimaging unit 20A is displayed, magnetic north measured by the triaxialmagnetometer 252 is displayed by an icon of a compass (reference sign44). Further, a message “Adjust the orientation of the front camera tomagnetic north” appears thereunder (reference sign 46). When the userchanges the orientation of the multiview imaging part 10 by rotating themain body part 30 of the panoramic imaging device 100, the shootingdirection of the imaging unit 20A changes, thus changing the orientationof magnetic north of the compass.

The user rotates the main body part 30 of the panoramic imaging device100 in accordance with the displayed message until magnetic north of thecompass represented by reference sign 44 points straight up, thuschanging the orientation of the imaging unit 20A.

FIGS. 36( a) and 36(b) illustrate the state when the orientation of theimaging unit 20A, a front camera, is aligned with magnetic northmeasured by the triaxial magnetometer 252. As illustrated in FIG. 36(a), the orientation of the imaging unit 20A is aligned with magneticnorth. At this time, magnetic north of the compass points straight up(reference sign 44), and a message “Good!” appears (reference sign 46)as illustrated in FIG. 36( b).

As described above, it is possible for the user to set the orientationof the main body part 30 of the panoramic imaging device 100 so that theshooting direction of the imaging unit 20A, a front camera, is set tomagnetic north. As a result, the center of the stitched panoramic imageis aligned with true north. The correspondence between the shootingdirection of the front camera and the direction of the center of thepanoramic image becomes clear by using the imaging unit 20A located onthe rear side of the operation/display unit 40 as a front camera andsetting the orientation of the front camera to magnetic north. It shouldbe noted that the panoramic imaging device 100 also incorporates thetriaxial acceleration sensor 254. This makes it possible to detect theinclination of the multiview imaging part 10 without using a spiritlevel and correct the multiview imaging part 10 so that it is levelduring the stitching process.

In the description given above, the shooting direction of the imagingunit 20A, a front camera, is set to magnetic north. However, theshooting direction of the imaging unit 20A may be set to an arbitraryangle with which the center of the panoramic image should be desirablyaligned. Alternatively, the shooting direction of the imaging unit otherthan the front camera may be set to magnetic north or an arbitraryangle.

FIGS. 37( a) and 37(b) and FIGS. 38( a) and 38(b) are diagramsdescribing a user interface adapted to set the shooting direction of thespecific imaging unit to magnetic north.

Assume that magnetic north measured by the triaxial magnetometer 252 isin the direction indicated by an arrow in FIG. 37( a). Here, of theimaging units 20A to 20E, the imaging unit 20C is oriented in thedirection closest to magnetic north.

FIG. 37( b) illustrates a screen displayed on the operation/display unit40. Images shot by the imaging units 20A to 20F are displayedrespectively in the areas A to F (reference sign 42). Because theimaging unit 20C is oriented in the direction closest to magnetic north,a compass is displayed in the area C where the image captured by theimaging unit 20C appears, thus indicating the direction of magneticnorth (reference sign 44). A message “Adjust the orientation of thecamera to magnetic north” appears under the areas A to F (reference sign46).

The user rotates the main body part 30 of the panoramic imaging device100 in accordance with the displayed message until magnetic north of thecompass represented by reference sign 44 points straight up, thuschanging the orientation of the imaging unit 20C.

FIGS. 38( a) and 38(b) illustrate the state when the orientation of theimaging unit 20C is aligned with magnetic north measured by the triaxialmagnetometer 252. As illustrated in FIG. 38( a), the orientation of theimaging unit 20C is aligned with magnetic north. At this time, magneticnorth of the compass points straight up (reference sign 44), and amessage “Good!” appears (reference sign 46) as illustrated in FIG. 38(b).

As described above, it is possible for the user to set the orientationof the main body part 30 of the panoramic imaging device 100 so that theshooting direction of the specific imaging unit is set to magneticnorth. It may be difficult, depending on the position of thephotographer, to shoot with the shooting direction of the imaging unit20A, located on the opposite side of the operation/display unit 40, setto magnetic north. In such a case, it is only necessary to set theshooting direction of any one of the imaging units other than the frontcamera to magnetic north.

Thus, the embodiment of the present invention has been described above.It should be understood by those skilled in the art that the aboveembodiment is illustrative, that the combination of components andprocesses can be modified in various ways, and that such modificationexamples also fall within the scope of the present invention.

A description has been given above of the panoramic imaging device 100that includes the multiview imaging part 10 and the main body part 30 asan example of a multiview imaging device. The multiview imaging part 10includes the plurality of imaging units 20A to 20F that are orientatedin different shooting directions. The main body part 30 includes theoperation/display unit 40. However, at least some of the plurality ofimaging units of the multiview imaging part 10 may be oriented in thesame direction. In particular, if the multiview imaging part 10 includesat least two imaging units that are oriented in the same direction, andif the positions of the viewpoints of these two imaging units aredifferent, it is possible to shoot an image having parallax. Usingparallax images allows to find the depth. As a result, athree-dimensional image can be generated. If a parallax image is apanoramic image, it is possible to generate a three-dimensionalpanoramic image. Such a stereo imaging device may be configured asanother example of a multiview imaging device, and it should beunderstood by those skilled in the art that the various characteristiccomponents and processes of the above embodiment are also applicable tosuch a stereo imaging device.

A description has been given above of the present embodiment by taking,as an example of a panoramic image, an entire celestial sphere panoramicimage. However, a panoramic image need not be an entire celestial spherepanoramic image. Instead, a panoramic image may be an image obtained bysynthesizing a plurality of images shot by a plurality of cameras whoseshooting directions are different.

It should be noted that the term “panoramic image” in the presentspecification is not limited to a “panoramic image” in a narrow sense,that is, a landscape or portrait image, 360-degree whole view image, andso on, but refers simply to an image that spans a wide area. Further, adescription has been given of an example of generating a panoramic imageas a synthesized image in the present embodiment. However, an outputsynthesized image need not be a so-called panoramic image. The presentinvention is applicable even when a synthesized image is an ordinaryimage of an arbitrary size. Alternatively, an output synthesized imagemay be an image made up of layered images having a plurality ofdifferent resolutions. Such a layered image may be formed so that whenan area of the image is enlarged, the enlarged area is replaced by animage with a higher resolution.

REFERENCE SIGNS LIST

-   10 Multiview imaging part-   12 Camera enclosure-   14 Fastening member-   20A to 20F Imaging unit-   30 Main body part-   40 Operation/display unit-   50 Tripod seat fastening hole-   70 Movie recording start/stop button-   72 Still image shooting button-   74 Auto exposure lock button-   75 Camera freeze button-   76 Auto white balance lock button-   77 Grouping button-   78 Extension control button-   80 Tripod-   100 Panoramic imaging device-   200 Panoramic movie stream generator-   210 Shooting control section-   220A to 220F Frame memory-   230 Movie stream multiplexing section-   232 Motion detection portion-   234 Mode determination portion-   236 Control signal generation portion-   240 Panoramic movie storage section-   242 Panoramic stitching section-   246 Switching section-   250 User interface section-   252 Triaxial magnetometer-   254 Triaxial acceleration sensor-   260 Display control section-   270 Touch panel control section-   280 Main processor-   290 Secondary storage device-   300 Bus interface/camera control interface-   310 Pass-through mask register-   320 Control data pass-through logic-   400 Exposure setting section-   410 Panoramic image storage section-   420 HDR synthesis section-   430A to 430F Frame buffer

INDUSTRIAL APPLICABILITY

The present invention is applicable to a technology adapted to capture asynthesized image.

1. An image generation device comprising: a plurality of frame memoriesadapted to store captured images from a plurality of imaging units; anda multiplexing section adapted to read the captured images of theimaging units from the frame memories and multiplex the captured imagesto generate an image stream, wherein the multiplexing section includes amotion detection portion adapted to detect a motion of a captured imageof at least one of the imaging units, a mode determination portionadapted to set, in accordance with at least either an amount of themotion detected by the motion detection portion or a power consumptionlimit of the image generation device, one of two modes, one foroperating the plurality of imaging units simultaneously and the otherfor operating the plurality of imaging units intermittently, and acontrol signal generation portion adapted to supply, to the plurality offrame memories, a control signal adapted to control reading of thecaptured images from the plurality of frame memories in accordance withthe mode set by the mode determination portion.
 2. The image generationdevice of claim 1 further comprising a shooting control section adaptedto control operations of the plurality of imaging units, wherein theshooting control section supplies, to the plurality of imaging units, acontrol signal adapted to operate the plurality of imaging unitssimultaneously or intermittently in accordance with the mode set by themode determination portion.
 3. The image generation device of claim 2,wherein, even when the mode for operating the plurality of imaging unitsintermittently is set by the mode determination portion, the controlsignal generation portion supplies, to the plurality of frame memories,a control signal adapted to operate the plurality of frame memoriesintermittently while operating the plurality of imaging unitssimultaneously.
 4. The image generation device of claim 1, wherein themode determination portion sets the mode for operating the plurality ofimaging units simultaneously if the amount of the motion detected by themotion detection portion is equal to or greater than a given threshold,and the mode determination portion sets the mode for operating theplurality of imaging units intermittently if the amount of the motiondetected by the motion detection portion is smaller than the giventhreshold.
 5. The image generation device of claim 4, wherein, even ifthe amount of the motion detected by the motion detection portion isequal to or greater than the given threshold, but if the imagegeneration device is not operating within the power consumption limit,the mode determination portion sets the mode for operating the pluralityof imaging units intermittently, and if the amount of the motiondetected by the motion detection portion is equal to or greater than thegiven threshold, and if the image generation device is operating withinthe power consumption limit, the mode determination portion sets themode for operating the plurality of imaging units simultaneously.
 6. Theimage generation device of claim 1, wherein, if the image generationdevice is operating within the power consumption limit, the modedetermination portion sets the mode for operating the plurality ofimaging units simultaneously, and if the image generation device is notoperating within the power consumption limit, the mode determinationportion sets the mode for operating the plurality of imaging unitsintermittently.
 7. The image generation device of claim 1, wherein, ifthe image generation device is operating on power from an external powersource, the mode determination portion sets the mode for operating theplurality of imaging units simultaneously, and if the image generationdevice is operating on power from a built-in chargeable battery, themode determination portion sets the mode for operating the plurality ofimaging units intermittently.
 8. An image generation method comprising:storing captured images from a plurality of imaging units in a pluralityof frame memories; and reading the captured images of the imaging unitsfrom the frame memories and multiplexing the captured images to generatean image stream, wherein the reading includes detecting a motion of acaptured image of at least one of the imaging units, setting, inaccordance with at least either an amount of the detected motion or apower consumption limit, one of two modes, one for operating theplurality of imaging units simultaneously and the other for operatingthe plurality of imaging units intermittently, and supplying, to theplurality of frame memories, a control signal adapted to control readingof the captured images from the plurality of frame memories inaccordance with the set mode.
 9. A non-transitory, computer-readablerecording medium storing a program for a computer, the programcomprising: a module configured to store captured images from aplurality of imaging units in a plurality of frame memories; and amodule configured to read the captured images of the imaging units fromthe frame memories and multiplexing the captured images to generate animage stream, wherein the module configured to read includes a moduleconfigured to detect a motion of a captured image of at least one of theimaging units, a module configured to set, in accordance with at leasteither an amount of the detected motion or a power consumption limit,one of two modes, one for operating the plurality of imaging unitssimultaneously and the other for operating the plurality of imagingunits intermittently, and a module configured to supply, to theplurality of frame memories, a control signal adapted to control readingof the captured images from the plurality of frame memories inaccordance with the set mode.